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EXPLORING THE EFFECT OF AFTERSCHOOL PARTICIPATION ON STUDENT’S 
COLLABORATION SKILLS, COMMUNICATION SKILLS, AND SELF-EFFICACY 



Denise Huang, Seth Leon, Cheri Hodson, Deborah La Torre, 

Nora Obregon, & Gwendelyn Rivera 
CRESST/University of California, Los Angeles 

Abstract 

This study addressed key questions about LA’s BEST afterschool students’ self-efficacy, 
collaboration, and communication skills. We compared student perceptions of their own 
21 st century skills to external outcome measures including the California Standardized Test 
(CST), attendance, and teacher ratings. We found a substantial relationship between student 
self-efficacy compared to student oral communication and collaboration skills. However, we 
did not find that higher attendance in LA’s BEST led to higher self-efficacy, though further 
investigation is needed. We found that LA’s BEST students were able to evaluate their 
abilities so that they arc similar to the outcome measures of CST and teacher ratings. 
Moreover, the high-attendance group demonstrated significantly better alignment with the 
teacher ratings than the lower attendance groups in self-efficacy, oral communication skills, 
and collaboration skills. 



CHAPTER I: 

INTRODUCTION 

Learning and Innovation skills are increasingly being 
recognized as the skills that separate students who are 
prepared for increasingly complex life and work environments 
in the 21 st century, and those who are not. A focus on 
creativity, critical thinking, communication and collaboration 
is essential to prepare students for the future. 

—P21 Framework 

On September 10 through 12, 2007, a poll of 800 registered voters across the country was 
conducted by Public Opinion Strategies and Peter D. Hart Research Associates on behalf of the 
Partnership for 21st Century Skills 1 The results revealed that an overwhelming 80% of voters 
believe that the skills students need to learn to be prepared for jobs in the 21st century are 
different from what they needed 20 years ago. While voters continue to believe that traditional, 



1 The Partnership for 21 st Century Skills is a coalition of leading education, business, and technology organizations. 
In 2002, they synthesized the perspectives of business, education, and government leaders to create a common 
language and strategic direction for efforts to infuse 21 st century skills into K-12 education in a report titled, 
“Learning for the 21 st century” (See http://www.21stcenturyskills.org). 
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basic skills are important, 66% of the voters believe that students need more than just basic 
reading, writing, math, and science. Furthermore, virtually all of those polled (99%) stated that 
the teaching and learning of 21st century skills (i.e., critical thinking and problems solving skills, 
computer and technology skills, and communication and self-direction skills) are very or 
somewhat important to the future economic success of the United States (Partnership for 21st 
Century Skills, 2007). The 21st century skills partnership states, “Every aspect of our education 
system — Pre-K through 12, postsecondary and adult education, after-school and youth 
development, workforce development and training, and teacher preparation programs — must be 
aligned to prepare citizens with the 21st century skills they need to compete” (2009). 

There are evidences that high-quality afterschool programs can promote a range of 
competencies and skills for the 21st century (Partnership for 21st Century Skills, 2008). Through 
active participation in a wide range of activities, students can use their skills as they work with 
afterschool staff to solve problems, make decisions, and take on leadership responsibilities. For 
example, the 2005 CRESST study, “Examining the Effects of Academic Skills and Academic 
Enablers Taught at LA’s BEST on the Achievement of Student Participants,” found that students 
and staff both perceived that interpersonal skills, motivation, and self-efficacy were promoted at 
LA’s BEST [Los Angeles’ Better Educated Students for Tomorrow afterschool program] (Huang 
et al., 2005). Interviews with LA’s BEST staff revealed that they used a variety of strategies to 
enhance these skills in students. For example, most staff (78.6%) indicated that they provided 
teamwork activities to improve students’ socialization skills, while 42.9% collaborated with 
students, and 14.3% taught conflict resolution skills. Nearly 93% of staff indicated that they give 
verbal encouragement to help students believe in themselves, and most staff (71.4%) reported 
that they try to make activities fun in order to promote motivation. The following year, the 
“Exploring the intellectual, social, and organizational capitals at LA’s BEST” study revealed that 
it was a social norm in the LA’s BEST setting for staff members to have high expectations for 
students to think critically and build their characters and for staff to offer caring support (Huang 
et al., 2007). Many staff members passionately described that they were sending encouraging 
messages to students about getting good grades, studying hard, behaving properly, going to 
college, being a positive community member, and becoming successful. 

Current literature supports the notion that there is a relationship between participation in 
afterschool programs and the development of 21st century skills; therefore, it is important to 
explore these claims and investigate whether participation in afterschool activities makes a 
difference in the development of these skills. This study intends to reduce the gap in literature by 
focusing on three of the 21st century skills that are commonly referenced and often associated 
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with afterschool activities: self-efficacy, collaboration skills, and oral communication skills. 
Accordingly, the main research questions for this study are as follows: 

1. Are LA’s BEST students’ feelings of self-efficacy associated with their collaboration 
and communication skills? 

2. Is there an association between participation in LA’s BEST and students’ feelings of 
self-efficacy, collaboration skills, and communication skills? 

3. How accurate are LA’s BEST students evaluating their own 21 st century skills as 
compared to external outcome measures of California Standardized Test (CST) results 
and teacher ratings? Are there differences in how students are evaluating themselves 
across the different attendance levels? 
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CHAPTER II: 



LITERATURE REVIEW 

Within the 21 st century framework, student competencies have moved beyond basic content 
knowledge and skills to include the attitudes, adaptability, and innovations that students will 
need to be successful in school, in the work place, and in relationships. This 21 st century 
framework shares many commonalities with the social cognitive theory on self-efficacy, which 
emphasizes the application of self-regulatory behaviors so that students can take purposeful, 
proactive, and reflective approaches in their own functioning (Zimmerman & Schunk, 2001). 

According to social cognitive theory, self-efficacy is the belief in one’s own ability to 
master a challenge or perform effectively (Bandura, 1982; Zimmerman & Kitsantas, 2005). It 
influences the types of behaviors that one employs when deciding whether to take on a task, 
what choices to make, and how much effort and perseverance to apply towards that task 
(Bandura & Wood, 1989). It requires students to use self-regulatory procedures that serve as 
mediators between personal views (e.g., beliefs about success), behavior (e.g., engaging in a 
task, strategies used), contextual characteristics (e.g., feedback from a teacher, expectations from 
parents), and actual learning outcomes (Pintrich, 2004). Self-efficacious students undertake 
difficult and challenging tasks more often than inefficacious students (Zimmerman & Kitsantas, 
2005), expand more effort, persist longer (Multon, Brown, & Lent, 1991), and have fewer 
adverse emotional reactions when they encounter difficulties (Pajares & Kranzler, 1995; 
Bandura, 1997). Self-efficacy also provides students with a sense of agency, which motivates 
their learning by propelling the cyclical phases of self-regulatory processes such as self- 
monitoring, self-evaluation, and self-reaction (Zimmerman, 2002). 

In social learning theory, self-monitoring refers to the cognitive tracking of personal 
functioning, such as keeping record of how much time one spends writing an essay. Self- 
evaluation refers to comparisons of how one performs against some standard, such as one’s prior 
performance or personal goals. Self-reaction refers to feelings of self-satisfaction regarding one’s 
performance. Increases in self-satisfaction enhance motivation, whereas decreases in self- 
satisfaction undermine future efforts to learn, thus influencing future goal setting (Schunk, 
2001 ). 

Importance of Self-monitoring and Self-evaluation 

In today’s society, with rapidly changing information and communication technology, the 
way that knowledge is generated and transmitted is incessantly evolving. Students and citizens 
need to develop self-directive processes to become proactive in their own course of lifelong 
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learning. Researchers and teachers agree that it is important to start by empowering students with 
the self-awareness and the strategic knowledge that they would need to self-improve 
continuously (Boekaerts & Corno, 2005; Schunk, 2001). 

However, according to Dunning, Johnson, Ehrlinger, and Kruger (2003), high achievers 
tend to underestimate and report lower than their actual capabilities, whereas low achievers tend 
to overestimate and report higher than their actual capabilities. At the same time, teachers in the 
classrooms were also aware of these shortfalls in students’ abilities to self-evaluate, and they 
emphasized the importance of training students to be able to gauge their performance accurately 
(Stone & May, 2002). Similarly, Dunning et al. also stressed that since individuals must be 
aware of their weaknesses before they can improve them, it is essential that individuals be able to 
evaluate their own abilities and skills accurately. This study will examine whether experiences in 
LA’s BEST provide opportunities for students to gain self-awareness and strategic knowledge 
for learning. 



How LA’s BEST Promotes Self-efficacy 

According to Bandura (1982), self-efficacy is acquired in a social learning environment in 
four ways: through performance attainment, by vicariously observing the experiences of others, 
by verbal persuasion by influential persons and allies, and by experiencing physiological states 
that are associated with self-appraisal across various situations. With the support of its caring 
staff members, LA’s BEST can provide these conditions in the following ways: 

• Performance attainment can be promoted by offering activities that foster students’ 
confidence and perceived competence (e.g., performances in arts, science projects, and 
physical activities). 

• Vicarious observation can be achieved by offering students positive experiences and 
interactions with accomplished individuals at the program who share commonalities 
with the students (e.g., afterschool personnel who are attending a four-year college). 

• Verbal persuasion can take place when staff offer encouragement and appraisal of the 
students and of teacher/parent expectations. 

• Since physiological states such as anxiety, stress, fatigue, and mood also contribute to 
efficacy beliefs, LA’s BEST can also offer opportunities for students to practice 
regulating these physiological states and to improve their cognitive self-appraisal by 
allocating time and space for challenging activities (e.g., public speaking or competitive 
sports). 

In this study, self-efficacy is defined as the capacity for personal management as well as 
the ability to process the skills and personal attributes that enable students to become successful 
adults. These skills and attributes include positive self-esteem, the ability to understand the 
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importance of learning and knowledge, the ability and skills to learn and gain knowledge, a 
belief in one’s own efforts, the ability to adapt, and the ability to analyze and problem solve. 

Self-efficacy is also related to communication and collaboration (Jerald, 2009). During the 
process of learning, self-regulated students maintain an active and ongoing awareness of task 
demands, of the effectiveness of their learning strategies, and of the progress towards the goals 
they have set (Pintrich, 2004). Self-efficacious students are also effective at seeking help, group 
management, and other aspects of collaboration and communication (Newman, 2008). Together 
these two competencies enable students to express and understand messages with accuracy and 
to work well together as part of a team. 

Oral Communication Skills 

Communication generally implies the exchange of thoughts and ideas with the intention of 
conveying information. Since communication involves almost every aspect of our interactions 
with other people, it provides the basis of how relationships are constructed and maintained. 
Good communication skills can reduce misunderstandings, errors, frustrations, and conflicts on a 
regular basis. Through communication, our ideas and interests are transmitted to other people; 
thus, the way we communicate serves as the foundation on which people form their opinions 
about us (Butler & Stevens, 1997). Therefore, effective communication leads to healthy personal 
and work relationships (Boyd, Lilling, & Lyon, 2007). According to the critical skills survey 
conducted by the American Management Association (AMA) in 2010, recruiters from major 
companies cited communication skills as a key factor in choosing managers. Thus, 
communication is a fundamental life skill that ought to be developed from an early age 
(Cameron & Quinn, 1999). 

Although there are many forms of communication, in this exploratory study we focus on 
oral communication. Competency in oral communication is defined as the ability to make clear 
and convincing oral presentations to individuals or groups, listen effectively and clarify 
information as needed, and facilitate an open exchange of ideas. 

LA’s BEST fosters these skills by having students read stories to each other, participate in 
discussions on topics that interest them, practice debating and clarifying their points of view in 
discussion/planning groups. Students may also communicate their ideas when they collaborate 
with each other, present in front of class or in shows, or join the debate teams, etc. 



2 The American Management Association is a world leader in talent development, advancing the skills of individuals 
to drive business success. Organizations worldwide, including the majority of the Fortune 500, turn to AMA as their 
trusted partner in professional development. 
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These specific examples point out that in many instances, development of communication 
and collaboration skills go hand-in-hand, especially when the students work in teams. The 
following section reviews the current literature on collaboration skills. 

Collaboration Skills 

Many researchers identify collaboration and oral communication skills as instrumental 
factors in supporting student learning because they provide students with opportunities to express 
ideas, share thoughts, and explain and help each other (Kafai, 2002). Moreover, in today’s global 
economy, we have shifted from a document-focused work style to a people-focused work style. 
Together with communication skills, working in teams is cited as another key factor in choosing 
managers (AMA survey, 2010). As such, the abilities to communicate effectively and work 
collaboratively with diverse groups of people are essential skills in the 21st century economy. 

Collaboration can be broadly defined as building and effectively utilizing relationships and 
informal networks to achieve a common goal (Roschelle & Teasley, 1995). It enables people to 
build on each other’s ideas and prior knowledge, resulting in innovations. Researchers tend to 
agree that collaboration can foster learning and productivity (Jarboe, 1996; Kumpulainen & 
Mutanen, 2000). As reported by Webb and Mastergeorge (2003), collaborative learning is 
considered an essential part of primary school instruction in countries such as the United States, 
Australia, and Israel. Johnson and Johnson (2000) state further that collaborative learning is 
widely used in educational settings, starting as early as pre-school, in afterschool programs, and 
extending to graduate schools. 

However, simply assigning students to small groups and asking them to work together does 
not necessarily promote collaboration (Knight & Bohlmeyer, 1990). A deliberate attempt has to 
be made to teach students how to collaborate effectively. Skills that students need to develop in 
order to collaborate efficiently would include management of group dynamics, problem-solving 
processes, and interpersonal communication skills (Webb & Farivar, 1994). Three types of 
communication skills are found to be especially important in collaborations: Students in 
successful collaborative groups were found to provide explanations, ask questions, and engage in 
argumentative discussions more often than students from less effective groups (Chan, 2001; 
Okada & Simon, 1997; Van Boxtel, 2000). As suggested in the previous examples, LA’s BEST 
provides the ideal setting to develop these oral communication skills. In addition, LA’s BEST 
also provides students with activities to develop collaboration skills such as conflict resolution. 
Furthermore, the program offers many opportunities to engage in hands-on, experiential 
activities that require teamwork and collaboration, such as team projects for a science fair. 
Consequently, the following collaborative skills are fostered: 
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• the ability to work effectively and respectfully with diverse teams, 

• a flexibility and willingness to be helpful in making necessary compromises to 
accomplish a common goal, and 

• the assumption of shared responsibility for collaborative work as well as the value of 
individual contributions made by each team member. 

In this study, collaboration skills are defined as having the ability to know and understand 
group dynamics, having the capacity of functioning productively in a team or group, 
understanding and maintaining group ethics, being able to absorb and handle pressure, and 
managing disputes among team members. 

The LA’s BEST Program 

Los Angeles’ Better Educated Students for Tomorrow (LA’s BEST) was first implemented 
in the fall of 1988. The program is under the auspices of the mayor of Los Angeles, the 
superintendent of the Los Angeles Unified School District (LAUSD), a board of directors, and an 
advisory board consisting of leaders from business, labor, government, education, and the 
community. 

LA’s BEST seeks to provide a safe haven for at-risk students in neighborhoods where gang 
violence, drugs, and other types of anti-social behaviors are common. The program is housed at 
selected LAUSD elementary schools and is designed for students in kindergarten through 
fifth/sixth grade. The LA’s BEST sites are chosen based on certain criteria, such as low 
academic performance and their location in low-income, high-crime neighborhoods. 

LA’s BEST is a free program open to all students in the selected sites on a first come, first 
served basis. Students who sign up for the program are expected to attend five days a week in 
order to reap the full benefits of the program offerings. Currently, the LA’s BEST program is 
implemented in 180 schools throughout LAUSD. LA’s BEST serves a student population of 
approximately 30,000 with about 80% Hispanic and about 12% Black elementary students. 
English Learners comprise at least half of the student population from most sites. Of this 
population, the majority’s primary language is Spanish, while the other percentage of the English 
Learner population is composed of those whose first language is of Asian/Pacific origin. 

Program Offerings 

Since its inception in 1988, LA’s BEST has adapted and updated its goals in response to 
educational policies, research, and theory. Over the years, the program has moved past its initial 
emphasis on providing a safe environment and educational enrichment to an emphasis on the 
development of the whole-child. In developmental theory, a whole-child curriculum is one that 
cultivates the development of students’ intellectual, social, and emotional well-being so that 
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children can achieve their full potential (Hodgkinson, 2006; Schaps, 2006). There are three and a 
half activity periods, or beats, at LA’s BEST: cognitive beat, homework beat, recreational beat, 
and snack (the half beat). These activity periods focus on the whole-child by emphasizing 
students’ intellectual, social-emotional, and physical development. 

The cognitive and homework beats focus on intellectual development. They are designed to 
develop the following traits: 

• Responsibility and positive work habits. LA’s BEST staff emphasize the importance 
of completing assignments, teach learning strategies and study skills, and provide a 
learning climate that reinforces positive attitudes towards school. 

• Love of learning. By encouraging active participation, explorations, and engaging 
research-based activities, the LA’s BEST program is crafted to instill a love of learning 
in its students. 

• Self-efficacy. Self-efficacy is achieved at LA’s BEST through guided experiences, 
challenging activities, and relationship building between staff and students. 

• Future aspirations. LA’s BEST is designed to help students develop aspirations for 
their futures; more specifically, the staff’s high expectations, the activities that build 
self-reliance, the value placed on education at the program, and the opportunities for 
collaboration and critical thinking are all meant to foster student aspiration. 

The recreational beat focuses on physical and social-emotional development. It is designed 
to develop the following benefits and characteristics: 

• Sense of safety & security. This is accomplished by providing students with a safe and 
nurturing environment. 

• Healthy lifestyle. This is accomplished through curriculum and activities that promote 
drug and gang prevention, healthy eating habits, and plenty of exercise. 

• Social competence. LA’s BEST aims to develop social competence by demonstrating 
respect for self and others and by providing students with opportunities to form 
friendships and develop trust and respect with peers and adults. 

• Sense of community. LA’s BEST fosters a sense of community by providing students 
with opportunities to participate in community- sponsored events, volunteer in 
community assignments, and go on field trips to local business and organizations. 

• Respect for diversity. A respect for diversity is taught at LA’s BEST using role 
modeling and a curriculum that enhances awareness. 

To summarize, the mission of LA’s BEST is to provide engaging settings so that each 
student learns in an intellectually challenging environment that is physically and emotionally 
safe for both students and adults. Furthermore, in the LA’s BEST environment, each student can 
be actively engaged in learning activities that are connected to their school and broader 
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community, and most importantly, each student has access to extra-curricular activities, 
academic enhancements, and qualified, caring adults. 



The Current Study 

In this study, it is hypothesized that attending LA’s BEST in higher intensity will give 
students opportunities to participate in activities that will enhance their self-efficacy, thus 
increasing their skills in regulating their own learning process. When students have the capacity 
to self-manage and the ability to navigate through the courses of their learning, they will become 
more successful in oral communication skills and collaboration skills. Figure 1 displays the 
hypothesized path model. 




In light of the important role of self-evaluation in the self-regulatory model, this study will 
take a second step to examine how well the students evaluate their own abilities when compared 
to external outcome measures. Since previous literature has associated self-efficacy with 
achievement outcomes (Taboada, Tonks, Wigfield, & Guthrie, 2009), California Standards Test 
(CST) scores on Math and English language arts are used as the outcome measure of academic 
achievement. In addition, teachers’ ratings of students’ citizenship are also used as a secondary 
outcome measure. It is hypothesized that each component of the students’ 21 st century skills (i.e., 
self-efficacy, oral communication skills, and collaboration skills) will be associated with the 
external outcomes measures: academic performance and teacher ratings of students’ citizenship 
and study skills. Multiple group modeling is used to test these hypotheses. An example of the 
hypothesized model is shown in Figure 2. 
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Figure 2. Hypothesized multiple-group model for self-evaluation. 
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CHAPTER III: 



STUDY DESIGN AND METHODOLOGY 

This chapter provides descriptions of the design, procedures, and methodology for the 
study. 

Study Design 

This study employs a quasi-experimental design that consists of student surveys and 
analysis of existing data. A survey was developed and administered to students from a sample of 
35 schools that host the LA’s BEST program. Each student in the sample completed the survey 
one time during the 2008-09 or 2009-10 school year. Students’ demographics, academic data, 
and school and afterschool attendance records were provided to the CRESST research team in 
May 2010. Since random assignment was not possible for this study, a propensity-based 
weighting method was used to minimize existing differences in student background 
characteristics. Structural Equation Modeling (SEM) was used to analyze the relationships 
between the latent factors. 

Site Selection and Recruitment of Participants 

Due to the nature and complexity of the student survey, the selection of sites was limited to 
those schools that served the upper elementary grade levels (4 th and 5 th grade) during the 2008- 
09 school year. This resulted in a population of 174 LA’s BEST sites. The study team then used 
a random number generator in order to select the sites and their alternates for recruitment. 

After the sites were identified, the research team obtained permission from the University 
of California, Los Angeles, Office for Protection of Research Subjects to implement the study 
procedures and instruments (approved on March 4, 2010). Upon approval, the research staff 
recruited 35 sites, obtaining written permission from the school principals to conduct the student 
survey administration. Finally, afterschool staff and school administrators helped the research 
staff distribute parent permission forms and student assent forms. 

The key participants in this study were LA’s BEST students and their classmates who were 
not enrolled in the LA’s BEST program at the time. During the first year of data collection, 
students in both 4 th and 5 th grade were eligible to participate. In order to maintain the same 
cohort, data collection was further limited to 5 th grade students during the second year of data 
collection. Overall, 911 students from the 35 sites completed the student survey. Of these 
students, 40 were excluded because of a lack of a student identifier (in order to link their survey 
and existing data), 8 were excluded because they failed to respond to five or more survey 
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questions, and 150 were excluded because they did not have demographic and performance data 
for the 2008-09 school year. In addition, 16 students were excluded because it was determined 
that they were not in 4 th or 5 th grade at the time of their survey. This resulted in a final analytic 
sample of 697 students. The descriptive characteristics of the students in the final analytic 
sample are shown in Table 1. 

Table 1 



Descriptive Characteristics of the Student Participants 



Characteristics 


Percent of total (n = 697) 


Language classification 


English only 


25.0 


Initially fluent 


16.5 


RFEP 


27.1 


ELL 


31.4 


Grade level (08-09 School Year ) 


4th grade 


73.6 


5th grade 


26.4 


Gender 


Male 


43.8 


Female 


56.2 


Parent Education 


Some college 


17.5 


HS grad 


21.1 


LT HS 


33.3 


Dec lined-unknown 


28.1 


Survey Administration Date 


Year 1 


54.1 


Year 2 


45.9 



Note. ELL = English language learner; HS = High School; LT HS = Less than high school; 
RLEP = Redesignated fluent English proficient. 



3 Expectation-Maximization algorithm (a method for finding maximum likelihood estimates of parameters in 
statistical models, where the model depends on unobserved latent variables) was employed to impute missing values 
for the 82 surveys with five or fewer missing items. 
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Measures 



CRESST employed the following measures to examine the relationship between 
participation in LA’s BEST and the development of 21 st century skills. A description of each 
measure is provided below. 

Student Survey 

The student survey was developed to examine students’ 21 st century skills (represented by 
self-efficacy, oral communication skills, and collaboration skills in this study). The survey 
included scales on self-efficacy, oral communication skills, and collaboration skills. Items for 
each of these scales were selected or adapted from previously established scales including 
Form A of the Communication Attitude Test (Bruten, 1985), the Student Attitudes toward Group 
Environment questionnaire (Kouros & Abrami, 2006), and Bandura’s Children’s Self-Efficacy 
Scale (2006). The questions selected from the first two scales focused respectively on the issues 
of interpersonal and personal affect about communication, as well as peer interaction, process, 
and product. Those questions selected from Bandura focused on the issues of perseverance, self- 
regulation, academic self-efficacy, and other self-efficacy issues (i.e., expectations, 
relationships). In addition, the survey included three background questions concerning students’ 
current and past participation in afterschool programs. 

Studies on the effects of “framing” (on questionnaires) often suggest a significant 
"response acquiescence bias" towards the positively framed items (Gamliel & Peer, 2006; 
Swamy, 2007). To minimize this bias, this study follows Guyatt et al.’s suggestion that 
questionnaires assessing attitudes toward educational programs include a mix of positively and 
negatively stated items (1999). All items, except the background questions, were asked using a 
four-item Likert scale consisting of “True,” “Mostly True,” “Mostly False,” and “False.” 

Following the pilot study at two LA’s BEST sites during Spring 2010, five questions were 
deleted and others were refined in order to improve readability and reliability. Cronbach’s alpha 
for the 24 items concerning oral communication (a = 0.807) and the 25 items concerning self- 
efficacy (a = 0.812) showed that both scales have good internal consistency. The collaboration 
scale was administered using 22 items. Following data collection, the item, “I get upset when 
kids in my group say bad thinks about my work” exhibited a poor negative item to total scale 
correlation (r = -0.199) and was deleted from final analysis. The research team believed that the 
wording of the item might have confused the students. With this item removed, the remaining 21 
items concerning collaboration showed good internal consistency (a = 0.831). 
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Existing Student Records 

LA’s BEST provided the research team with afterschool attendance data. In addition, 
LA USD provided matched data on student background characteristics, achievement, and report 
card data. 

Achievement. LAUSD provided student-level data including the test scores from the CSTs 
for math and English language. These test scores were used in this study as outcome measures 
for academic performance (AP) to gauge students’ self-evaluation. Academic performance was 
employed as an outcome measure because self-efficacy, oral communication skills, collaboration 
skills, and self-regulation skills are all consistently linked to school performance. 

According to Zimmerman, Bandura, and Martinez-Pons (1992), students’ self-efficacy and 
self-regulation of academic performance is linked to their awareness of covert and overt 
outcomes of their behavioral functioning. Self-regulated students are significantly more likely 
than non- self-regulated students to know how well they did on a test before it was graded by 
their teachers. Furthermore, self-efficacy, goal setting, self-monitoring, use of learning strategies, 
and self-reflection all have consistently been shown to be good predictors of academic outcomes 
such as math and writing performance and course grades (Cleary, 2006; Graham, Harris, & 
Troia, 2000; Pajares & Urdan, 2006; Schunk, Pintrich, & Meece, 2008; Zimmerman & Martinez- 
Pons, 1990). They have also been shown to be predictors of adaptive behaviors such as 
persistence, resiliency, and effort (Bandura, 1997; Schunk, 1983). 

Similarly, Kastner, Raggio, and May (2000) found that language-based skills are associated 
with future academic performance in school-aged students, while Hughes & Large (1993) found 
that oral communication skills are positively correlated with college pharmacology students’ 
grades on theory papers and long essays. Meanwhile, students with effective collaboration skills 
are found to exercise more autonomy with their learning and obtain higher learning outcomes 
than peers less efficient in collaboration skills (Gillies & Ashman, 1998). 

At the same time, research has also shown that teachers can easily identify students who 
are self-regulated by their attributes (Zimmerman, & Martinez-Pons, 1988). For example, self- 
starters who display perseverance on learning tasks; students who are confident, strategic, and 
resourceful in overcoming problems; and students who are self-reactive to task outcomes are 
frequently identified by their teachers as self-regulated. Thus, teacher ratings on students’ study 
skills and citizenships are also used as outcome measures to gauge students’ self-evaluation 
ability on their survey responses. 

Students’ work and study skill ratings. The student report card data included teacher 
ratings on five categories of their students’ work and study skills. These categories were rated 
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using a five-point Likert scale. The categories that the teachers rated included the following: 
“makes good use of time,” “completes work on time,” “organizes materials,” “presents neat and 
careful work,” and “works independently.” Each of these skills requires self-regulation. 

Zimmerman and Martinez-Pons (1988) stated that self-regulated students are distinguished 
by their sensitivity to (and resourcefulness when dealing with) the effects of the social and 
physical environment on their learning. They are more likely to organize or restructure their 
place of study, seek social assistance, keep records, and monitor their progress. Researchers have 
also indicated that time planning and management are significantly related to academic 
achievement (Pintrich, Smith, Garcia, & McKeachie, 1993; Zimmerman & Martinez-Pons, 
1986). Conversely, when students' self-regulatory skills are lacking, they will often be at high 
risk for exhibiting poor academic outcomes and maladaptive behaviors such as poor organization 
and time-management skills, inconsistent work completion and accuracy, and avoidance of 
school-related activities (Pintrich et al., 1993). These self-regulatory skills are all reflected in the 
teacher ratings of the study skills. Thus, these ratings were used as outcome measures to gauge 
the accuracy of students’ self-evaluation. 

Students’ citizenship ratings. The student report card data also included teacher ratings of 
nine student citizenship items. These items were rated using a four-point Likert scale, with “1” 
being lowest and “4” being highest. The numerical ratings indicated whether a student’s 
citizenship was (1) Poor, (2) Inconsistent, (3) Consistent, or (4) Strong. 

In order to create a closer match between the ratings and the factor of collaboration, the 
research team further separated the citizenship ratings into two subcomponents: students’ self- 
discipline and students’ peer interaction. The subcomponent of self-discipline included the 
following items: “follows direction,” “accepts and respects authority,” “shows dependability,” 
“takes responsibility,” and “exercises self-control.” Like the work and study skill ratings, these 
self-discipline ratings were used because they reflect a student’s ability to self-regulate. 

A recent study reported that self-discipline in students is a predictor of academic abilities 
(Ponitz, McClelland, Matthews, & Morrison, 2009). The researchers used the Head-Toes-Knees- 
Shoulders Task (HTKS) to evaluate 343 kindergartens’ ability to self-regulate. The HTKS task 
measures the ability to listen, remember instructions, and follow motor commands. The 
researchers concluded that students with higher levels of these self-regulation skills in the 
beginning of the school year achieved higher scores in vocabulary and math at the end of the 
school year. Self-regulated learners are also better at using self-control to avoid distraction from 
their learning tasks (Corno, 1993; Zimmerman et al., 1992). Lurthermore, as reported in a study 
on discourse and self-regulation, it was found that through open discussions, students get to 
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practice skills such as following directions, accepting and respecting authority, showing 
dependability, and taking responsibility (Mason, 1998). 

The items in the second subcomponent of the citizenship ratings, peer interaction, included 
teachers’ observations on students’ demonstration of social competency with peers. The possible 
observations were, “demonstrates appropriate social interaction with peers,” “demonstrates fair 
play,” “resolves conflicts appropriately,” and “cooperates well in group situations.” 

Research shows that social strengths can contribute to self-efficacy. Perceived social self- 
efficacy measures students’ beliefs in their own capabilities to form and maintain social 
relationships and manage different types of interpersonal conflicts (Bandura, 1994). A high sense 
of social efficacy promotes satisfying and supportive social relationships. It fosters social 
behaviors such as sharing, helpfulness, kindness, and cooperativeness, all of which build peer 
acceptance (Ladd, Price, & Hart, 1988). Students who are considerate of their peers and are 
accepted by them will find school to be a favorable environment (Bandura, 1994). Students who 
are comfortable seeking help from adults and peers are found to achieve higher mastery of their 
academic coursework than those who are not confident about their social capabilities (Newman, 
1991). Thus, the teacher ratings on citizenship can be used as outcome measures for social 
efficacy and collaboration. 

Meanwhile, evidences also suggest that oral communication and collaboration stimulate 
learning at school. A study in science discourse found that group discussions stimulated the 
construction of advanced knowledge by sharing ideas collaboratively. Students reasoned and 
argued through steps of opposition and construction (Mason, 1998). Through these procedures, 
students exercise study skills by organizing their thoughts, presenting neat and careful work, 
working independently as well as cooperatively, and taking responsibility. In open discussions, 
students also get to practice citizenship skills such as following direction, accepting and 
respecting authority, showing dependability, taking responsibility, and exercising self-control. In 
maintaining constructive discussions, students develop collaboration skills such as interacting 
appropriately with peers, playing fair, resolving conflicts appropriately, and cooperating in group 
settings. 

These indicators of self-efficacy, oral communication skills, and collaboration skills are all 
consistent with the teacher ratings of their students’ study skills, citizenship, and peer interaction. 
Thus, they were used as outcome measures to gauge the accuracy of students’ self-evaluation in 
these factors. 
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Data Collection Procedures 



Thirty-five LA’s BEST sites were randomly selected by the research team, and permission 
for the schools to participate in the study was granted by each principal. Students were each 
surveyed once during the period of the study. 

During the first year of data collection (2008-09), the sample was comprised of both 4 th 
and 5 th grade students. During this year, surveys were administered during school hours. The 
teachers were requested to survey all the students in their classrooms, including LA’s BEST and 
non-LA’s BEST participants. Because of increasing pressure concerning budget cuts and state 
test scores, school administrators and teachers expressed concern about taking students away 
from their normal class activities. As a result, it was more difficult to obtain parent permission 
and student assent than anticipated. In response, the research team and the LA’s BEST 
operations office extended data collection for a second year (2009-10). During this second cycle 
of data collection, data collection was limited to the afterschool hours. In addition, eligibility to 
participate was limited to the 5 th grade students (4 th graders during the first year) in order to 
maintain the original student population. It should be noted that the vast majority of schools that 
host LA’s BEST do not serve students in 6 th grade. Issues involving data collection were 
addressed in the methodology by controlling for the date of survey administration during 
analysis. 

Data Analysis 

The following describes the strategies and procedures used to establish the attendance 
categories and to analyze the quantitative data sources. 

Categorizing Attendance 

Examination of afterschool attendance patterns indicates that students participate in LA’s 
BEST with varying regularity. Therefore, it was necessary to set criterion to measure the 
intensity of attendance. Previous studies have indicated that students need to participate at least 
100 days in order to reap program benefits (Frankel & Daley, 2007; Huang, Leon, La Torre, & 
Mostafavi, 2008). Initially, the research team intended to use 100 days as a reference in setting 
the attendance criteria for this study. 

Defining attendance intensity. Although the average attendance of LA’s BEST students is 
114 days, the students who obtained informed consent and participated in the study showed an 
average attendance of 153 days. Due to this unexpected pattern, for the purpose of analysis, it 
was necessary to create categories for students with well over 100 days to ensure an adequate 
sample size under each category. Additionally, since the school teachers were not successful in 
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recruiting non-participants into the study, only a handful of non-LA’s BEST students had 
obtained parent consent to participate in the study. All these challenges resulted in a study 
sample very different than the one that the study team had intended to recruit. Alternately, the 
team took note of the number of days students attended during the 2008-09 school year and 
categorized attendance into the following three levels of intensity: 0-100 days (Level l), 4 101— 
170 days (Level 2), and 171-239 days (Level 3). Table 2 shows the distribution of the three 
levels across the samples. 



Table 2 

LA’s BEST Attendance Intensity 







Average LA’s BEST attendance intensity 
(2008-09) 






Level 1 
(0-100 days) 


Level 2 

(101-170 days) 


Level 3 

(171-239 days) 


Total 


Usable sample linked to attendance 
Analytical sample 


249 (29%) 
164 (24%) 


276 (32%) 
226 (32%) 


338 (39%) 
307 (44%) 


863 (100%) 
697(100%) 



Quantitative Data Analysis 

There were two primary steps involved in the research team’s analytic approach. The first 
step was to control for existing background differences between participants across the different 
levels of attendance intensity. After controlling for differences between participants, the research 
questions were then analyzed using a series of SEM models. 

Controlling for differences. In social science, randomized control experiments are often 
difficult to achieve due to study design and/or ethical issues; subsequently, quasi-experimental 
designs using propensity scoring methods have become the best approximate to a randomized 
control design and are gaining widespread use. In this study, since the students were not 
randomly assigned to the three attendance levels, it was necessary to control for existing 
differences in students’ background characteristics and other factors so that potential associations 
could be explored. The propensity scoring method was used to complete this task. To compute a 
propensity score, two iterations of logistic regression were used to estimate the probability that a 
student was in one of the three attendance intensity groups. 



4 Students who were not enrolled in LA’s BEST during the 2008-09 school year were included in Level 1 because 
the sample size was too small to study independently. 
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Logistic regression. Logistic regression was employed to model the relationship between 
student background characteristics and the likelihood of a student attending the LA’s BEST 
program at the Level 1 attendance intensity as compared to the Level 2 attendance intensity. This 
initial logistic regression produced estimates for the first two levels of LA’s BEST attendance 
intensity. 

Next, a similar process was used to estimate the likelihood of a student attending the LA’s 
BEST program at the Level 3 attendance intensity as compared to the Level 2 attendance 
intensity. A formula was then used to place the estimates for the students in the highest level of 
attendance category (Level 3) on the same scale as the estimates obtained from the initial logistic 
regression. This resulted in a propensity scalar that would create balance across each attendance 
intensity level so that none of the background indicators would be overrepresented in any of the 
three attendance levels. 

Weighting process. The propensity score was applied in a weighting approach to create 
balance among student background characteristics and other factors across intensity groups. 
Cases were inversely weighted relative to their propensity outcome and normalized so that the 
final weighted sample was the same size as the original unweighted sample. Once balance was 
reached among student background characteristics across the intensity levels, valid comparisons 
could be made. 

The list of the variables included in the balancing procedures along with a description of 
the before and after weighting procedure is shown in Appendix A. As indicated in the tables 
provided in Appendix A, there were no longer any variables that were significantly associated 
with LA’s BEST attendance intensity during the 2008-09 school year after applying the 
weighting process. A detailed example of how the propensity scalar was created is also presented 
in Appendix A. 

Structural equation models. Structural equation modeling was conducted using the EQS 
software to examine the relationships between LA’s BEST program attendance during the 2008- 
09 school year and student readiness to utilize 21 st century skills. Lirst, a path model was used to 
test the hypothesis that higher LA’s BEST attendance intensity would lead to higher self-efficacy 
and subsequently higher oral communication skills and collaboration skills (research questions 1 
and 2). Multiple group structural models were then applied to examine how accurate the 
students’ self-evaluations were as reflected by the strength of the association between the 
measures of 21 st century skills and those of student performance and teaching ratings across the 
three LA’s BEST attendance levels (research question 3). 
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CHAPTER IV: 



STUDENT DEMOGRAPHIC ANALYSIS AND SEM MODELING RESULTS 

This chapter presents the demographic analysis and modeling results for the study. 

Student Demographics 

Descriptive results of the students’ survey responses in relation to their background 
characteristics are shown in Table 3. On a four-point Likert scale, the mean response score was 
slightly over 3 for each of the 21 st century s ki ll measures. A score of 3 would indicate a “Mostly 
True” response to a positively framed question or a “Mostly False” response to a negatively 
framed question. 

With regard to language classification, English Language Learners tended to report 
somewhat lower self-efficacy and oral communication skills than English Only and Initially 
Eluent students. There were no significant differences in collaboration skills across the language 
classification categories. 

Fourth grade students reported slightly higher self-efficacy and collaboration skills than did 
fifth grade students. There were no significant differences in 21 st century skills reported by boys 
as compared to girls. It is interesting to note that students whose parents attended some college 
reported higher oral communication skills than those whose parents declined to provide 
education data. 

Table 3 



Student Demographic and Survey Responses 







Mean Response Score 




Self-efficacy 


Oral communication 


Collaboration 


Language classification 


** 


** 




English only 


3.18 


3.26 


3.05 


Initially fluent 


3.23 


3.32 


3.10 


RFEP 


3.17 


3.17 


3.14 


ELL 


3.04 


2.97 


3.08 


Grade Level 


** 




* 


4 th 


3.18 


3.14 


3.11 


5 th 


3.09 


3.18 


3.03 
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Mean Response Score 






Self-efficacy 


Oral communication 


Collaboration 


Gender 








Male 


3.13 


3.15 


3.06 


Female 


3.17 


3.15 


3.12 


Parent education 




** 




Some college 


3.20 


3.27 


3.20 


HS grad 


3.16 


3.17 


3.16 


LT HS 


3.14 


3.14 


3.15 


Declined-unknown 


3.12 


3.03 


3.12 


Combined mean 


3.15 


3.15 


3.09 



Note. ELL = English language learner; HS = High school; LT HS = Less than high school; 

RFEP = Redesignated fluent English proficient. 

** represents significance across the specific demographic categories atp<0.01, while * represents 
significance across the specific demographic categories at p<0.05. 

Structural Modeling Results 

First, results from the path model are presented. 

Path Model 

The path model demonstrated adequate fit as indicated by comparative fit index 
(CFI) = 0.961 and root square error of approximation (RMSEA) = 0.056. As shown in Figure 3, 
the three latent factors consist of student self-efficacy, oral communication skills, and 
collaboration skills. The shaded box of LA’s BEST attendance is a simple ordered indicator of 
the three LA’s BEST attendance levels. Significant associations are indicated by solid arrows 
with the corresponding standardized beta coefficients displayed inside the box. Standardized beta 
coefficients can be interpreted like correlation coefficients (r). A standard rule of thumb converts 
an r of 0.10 to a small effect size, while an r of 0.24 can be considered moderate, and an r of 
0.37 or greater can be considered a large effect. 

The first research question presented was, Are students’ feelings of self-efficacy associated 
with their collaboration and communication skillsl 

As expected and in support of literature on self-efficacy and 21 st century skills, significant 
positive findings are shown in the solid paths drawn from self-efficacy to collaboration skills 
(r = 0.61) and communication s ki lls (r = 0.75). As for the second research question (Is there an 
association between participation in LA’s BEST and students’ feelings of self-efficacy, 
collaboration skills, and communication skills?), the findings indicate that intensity of LA’s 
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BEST attendance is not significantly associated with student self-efficacy, oral communication 
skills, or collaboration s ki lls. 




Figure 3. The path model. 



There could be multiple reasons why significant associations are not found between LA’s 
BEST attendance intensity and the 21 st century skill measures. First, due to the challenges for 
data collection during this time of high economic and emotional turmoil, the study ended up with 
an overall reduced sample size largely represented by high- attending students. This resulted in 
the underrepresentation of LA’s BEST participants that are attending less than 100 days. In order 
to generate the sample size necessary for analysis, the study team had to combine students with 
some attendance (n = 92) and those with no attendance (n = 72) together. Although extreme care 
was applied in controlling students’ existing background differences, there could be unmeasured 
differences between students with some attendance compared to those with no attendance. This 
may make inferences regarding comparisons to this LA’s BEST attendance category more 
difficult. 

In addition, causal inferences are generally more constrained with cross-sectional analyses. 
For this study, since data were collected at one time point, even though students’ existing 
background differences were controlled, variations across the three attendance levels (in the 
baseline of 21 st century skill competencies) still may not be completely captured by the available 
background controlling variables. 

Furthermore, prior research has indicated that students with low skill levels sometimes 
overestimate their abilities while students with higher skill level tend to underestimate their 
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abilities, thus creating unbalanced bias for the accuracy of students’ self-evaluations (Kruger & 
Dunning, 1999). If this phenomenon is present in the way students responded, the results might 
have consequences to the findings of the study. 

To examine this phenomenon further and to proceed with the third research question (How 
accurate are LA ’s BEST students evaluating their own 21'' century skills as compared to external 
outcome measures of C ST results and teacher ratings? Are there differences in how students are 
evaluating themselves across the different attendance levels?), four structural models were 
constructed to test these relationships. All the model results are based on the weighted sample 
that controls for existing student differences. 

Multiple Group Structural Self-Evaluation Models 

Multiple group structural models are applied to examine the strength of association 
between measures of 21 st century skills and those of student performance and teacher ratings 
across the three LA’s BEST attendance levels. In other words, we might ask, “How accurate are 
students evaluating their own ability as compared to external outcome measures of CST results 
and teacher ratings? Are there differences in how students are evaluating themselves across the 
different attendance levels?” 

In order to conduct the multiple group structural models, all constraints have to be removed 
and model fit established. This was achieved through the following steps. First, a model was 
specified to constrain all model correlations to be equal across the three LA’s BEST attendance 
groups. A Lagrange multiplier (LM) test was then used to see if model fit would be significantly 
improved by releasing a constrained correlation (allowing it to vary across groups). When the 
test indicated a probability of less than 0.05, the constrained correlation was released. A released 
constraint indicated that the correlation (association) was different depending upon LA’s BEST 
attendance intensity level. This process was repeated until all remaining constraints were no 
longer significant and would not improve model fit by being released. The hypothesized model 
was tested for adequate fit as indicated by the comparative fit index (CFI) and the root mean- 
square error of approximation. In the case that adequate fit was not obtained, the LM test for 
adding parameters was used to re-specify the model. 

Figures 4, 5, 6, and 7 represent the findings. In these figures, each attendance level 
represents a group. Each rectangle contains the three correlations for Attendance Levels 1, 2, and 
3, respectively. Shaded rectangles indicate that significant differences in the correlations were 
observed across the three attendance groups. 

Self-efficacy. The first model examined the relationship between self-efficacy and outcome 
measures of academic performance and teacher ratings. Due to the sample size of the study, it 
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was necessary to combine the 25 self-efficacy items into eight item parcels to reduce the number 
of parameters in the model. Item parcels were defined by four self-efficacy subscales and, as 
suggested by the LM test, the framing of the item (positive/negative). This approach was applied 
so that potential differences in factor loadings across attendance levels would not be hidden or 
glossed over by the parcel combination. See Appendix B for a detailed representation of the 
measures in each multiple group model. 

In each of the following models, academic performance (AP) is represented by a factor that 
combines performance in CST scores from the 2008-09 school year in math and English 
language arts. Another factor combines teacher ratings (TRs) of students’ work, study habits, and 
citizenship skills. In Figure 4, a single factor of self-efficacy is modeled. 

The fit indices from this model indicated a less-than-adequate model fit (CFI > 0.894; 
RMSEA = 0.091). An examination of the EM test results suggested that parcels within the self- 
efficacy factor had different associations with AP and TR depending on how the item was 
framed. In order to improve model fit, LM tests suggested that two self-efficacy factors be 
included based on item framing to allow these associations to be represented. 




For this reason, the survey items from the self-efficacy scale were separated into two 
factors: positively (PSE) or negatively (NSE) framed. Examples of positively framed items are, 
“It is easy for me to finish my homework on time,” and “I am good at organizing my 
schoolwork.” After separation, there were 17 positively framed items with a Cronbach alpha of 
these items at 0.802. Examples of negatively framed items are, “It is hard for me to finish my 
assignments on time,” and “I am not good at taking notes during class.” There were eight 
negatively framed items; the Cronbach alpha for these items is 0.695. 

After re-specifying the model, fit indices indicated a substantially improved model fit 
(CFI = 0.978; RMSEA = 0.039). As shown in Figure 5, the positively framed self-efficacy factor 
had a modest association with both AP (r = 0.20) and TR (r = 0.12), and these associations were 
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not different across the LA’s BEST attendance levels. However, there were differences in the 
associations between the outcome measures and NSE across the LA’s BEST attendance levels. 
Specifically, students who attended LA’s BEST over 170 days had a considerably stronger 
association between the negatively framed self-efficacy factor and both AP (r = 0.55) and TR 
(r = 0.33). 





Figure 5. Self-efficacy model 2. 

Communication and student performance. The communication model examined the 
relationships between oral communication skills, student performance in math and English 
language arts (AP), and teacher ratings (TR). The model is shown in Figure 6. Item parcels were 
defined by two oral communication subscales (interpersonal and personal affect) and the framing 
of the item. Initial fit indices indicated an adequate model fit (CFI = 0.963; RMSEA = 0.071). 

The model reveals that oral communication skills were more strongly associated with AP 
than with TR. It was found that students who attended LA’s BEST over 170 days demonstrated 
stronger associations between oral communication questions and both AP and TR. There was no 
significant association between oral communication and TR for students who attended LA’s 
BEST less than 170 days, while students who attended LA’s BEST over 170 days had a small 
but significant association (r = 0.17). Students who attended LA’s BEST over 170 days had a 
strong association between oral communication and AP (r = 0.50). This indicates that students in 
the higher attendance group are better at evaluating their ability in oral communication skills, as 
affirmed by the alignment with the CST scores and teacher ratings. 
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Figure 6. Oral communication skills model. 



Collaboration and Citizenship. Figure 7 displays the collaboration model that the 
research team used to examine the relationships between a single factor of collaboration skills 
and AP and TR. For this model, item parcels were combined based on three subscales (peer- 
support, product of collaboration, and process of collaboration). 5 The fit indices indicated an 
adequate model fit (CFI = 0.973; RMSEA = 0.047). The findings indicate that there was a small 
but significant association between collaboration and AP regardless of LA’s BEST attendance. 
There were no differences in the association between collaboration and teacher ratings across the 
three attendance categories. Similar to the oral communication factor, there was no significant 
association between collaboration and TR for students who attended LA’s BEST less than 170 
days. However, for those who attended LA’s BEST over 170 days, a moderate association 
(r = 0.27) was present. A detailed representation of the measures in this model is presented in 
Appendix B . 




Figure 7. Student collaboration model 6. 



5 Item parcels were not created based on item framing as there were only three negatively framed items. 






CHAPTER V: 



DISCUSSION AND CONCLUSION 

The path model supported current and existing literature and Research Question 2, which 
suggests that self-efficacy is significantly related to both collaboration skills and oral 
communication skills. LA’s BEST students that rated themselves high on self-efficacy also had 
high oral communication skills and collaboration skills. However, the model failed to support the 
first research question and the hypothesis that higher attendance of LA’s BEST students will lead 
to higher self-efficacy, oral communication skills, and collaboration skills. It is found that 
attendance intensity at LA’s BEST has no relationship with self-efficacy. As expressed 
previously, there can be a number of plausible explanations for this occurrence, such as: 

• This is a student sample that represents more of the higher attendance group than the 
normal distribution of LA’s BEST participants; this unexpected occurrence may have 
skewed the study sample and, thus, the findings. 

• Since this is a cross-sectional study looking at student responses from a single time 
point, perhaps having a pre- and post-survey will be better at detecting student 
differences in their development of self-efficacy. 

• According to the Kruger & Dunning effect, it may also be possible that students higher 
in self-efficacy are rating themselves lower than their actual performance, whereas 
students lower in self-efficacy are inflating their abilities. 

Future studies can examine these issues further. 

Meanwhile, interesting findings emerged from this exploratory study. To address the 
emphasis on the importance of self-evaluation noted by the 21 st century framework, social 
cognitive studies, and the work of Kruger & Dunning, we turn to Research Question 3, which 
examines how LA’s BEST students are evaluating themselves across self-efficacy, oral 
communication skills, and collaboration skills. Specifically, the question inquires whether 
intensity of attendance contributes to their self-evaluation skills. This study found that higher 
attending LA’s BEST students consistently evaluate themselves more in accordance with the 
external measures than the lower attendees do. 

As an example, for self-efficacy, the positively framed factor had a modest association with 
both academic performance and teacher ratings; but these associations were not different across 
the LA’s BEST attendance levels, indicating that all three attendance groups evaluate themselves 
quite similarly with the external measures. However, the negatively framed factors appeared to 
have stronger associations with academic performance and teacher ratings than did the positively 
framed factor; moreover, these associations (between negatively framed factors and self- 



31 




efficacy) were also significantly different across the LA’s BEST attendance levels. Students who 
attended LA’s BEST over 170 days had the strongest association between the negatively framed 
self-efficacy factor and the external outcome measures of academic performance and teacher 
ratings. These findings suggest that, in this study, negatively framed self-efficacy items appeared 
to have better predictive power than the positively framed items. Furthermore, the findings also 
suggest that students in the higher attendance group can respond to the negatively phased items 
better than the lower attendance group and are better at evaluating their self-efficacy in 
alignment with their academic performance in CSTs and with their teachers’ ratings of 
themselves. 

From a different perspective, the negatively framed items in this study hinted at the 
students’ abilities to acknowledge that they are not “excellent” at some tasks. For example, one 
of the negatively framed items is, “I am not good at learning to read.” In order to come to terms 
with that self-assessment, some cognitive processes need to be taken. Studies such as the theory 
of planned behavior (Ajzen, 1991) and several theories of individual differences in motivation 
(Carver & Scheier, 2000) have indicated that factors such as message framing and dispositional 
motivations can be integrated into larger theories of behavior change. They propose that behavior 
is regulated by two distinct systems: an approach system that regulates appetitive behavior 
toward potential rewards and an avoidance system that regulates behavior away from potential 
threats or punishments. Similarly, studies on self-efficacy, motivation, and achievement also 
focused on the need for achievement and avoidance of failure (Elliot & Church, 1997; Elliot & 
McGregor, 2001). Need for achievement represents an approach- valenced motivational 
disposition to experience pride upon the demonstration of competence, while fear of failure is 
defined as a disposition to avoid failure in achievement settings since one felt shame upon failure 
(Elliot & Thrash, 2004). How these different attributions will play into students’ responses can 
be further examined in future studies. 

As for oral communication skills, similar to self-efficacy, students who attended LA’s 
BEST over 170 days again demonstrated stronger alignment with the way they are evaluating 
themselves in oral communication skills and the outcome measures of their academic 
performance and teacher ratings. For academic performance, even though all students’ ratings 
are significantly related to their CST performance, the higher attending group shows the 
strongest relationship. As for teacher ratings, the higher attending students are the only group 
that shows a significant relationship with their self-evaluation and their teachers’ ratings. 

For collaboration skills, teacher ratings were more in alignment with student evaluation; 
however, contrary to oral communications, all students’ self-evaluations were only marginally, 
but significantly, related to academic performance. It should also be noted that the higher 
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attending group was the only group that was significantly aligned with their teachers’ ratings in 
citizenship. 

These findings indicate that students’ self-evaluations on self-efficacy, oral communication 
skills, and collaboration skills are generally more aligned with their academic performance than 
with teacher ratings on citizenship. It should also be noted that the higher attending group’s self- 
evaluations are more strongly related to academic performance than the other attendance groups 
are. Meanwhile, teacher ratings on citizenship appeared to have more discriminatory power, as 
the high-attending students were the only group whose self-evaluations were significantly related 
with teacher ratings (other than positively framed items on self-efficacy). 

Regarding the third research question, LA’s BEST students appeared to be mostly accurate 
when evaluating themselves in self-efficacy, oral communication skills, and collaboration skills. 
The higher attending students have also consistently indicated that their self-evaluations are in 
closer alignment with both their academic performance and the teacher ratings. 

Conclusion & Recommendations 

In summary, this study supported the literature that associates self-efficacy with oral 
communication skills and collaboration skills. However, the study findings did not support the 
hypothesis that higher attendance in LA’s BEST will lead to higher self-efficacy, though further 
investigation into the self-evaluative abilities of the LA’s BEST students would provide new 
insights. In addressing the third research question, it is found that, overall, LA’s BEST students 
are able to evaluate their abilities so that they are similar to the outcome measures of CST 
performances and teacher ratings. Moreover, the high-attendance group demonstrated 
significantly better alignment with the teacher ratings than the lower attendance groups in self- 
efficacy, oral communication skills, and collaboration skills. 

Examinations on the literature and teachers’ practices underscored the importance of 
students employing self-regulation techniques. The social cognitive theory recommends that 
teachers help students develop self-regulation strategies and set realistic expectations for their 
academic accomplishments. It appears that LA’s BEST is promoting self-evaluation skills, 
particularly among the higher attendees. In the future, LA’s BEST can continue to improve all of 
its students’ self-evaluation skills by applying focused strategies and by making more intentional 
efforts. Research suggests that the simplest tools to encourage students’ self-appraisal are 
evaluative questions that force students to think about their own work. Some examples of these 
questions include the following: 

• How much time and effort did you put into this? 

• What do you think your strengths and weaknesses were in this assignment? 
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• How can you improve your assignment? 

• What are the most valuable things you learned from this assignment? 

In order to gauge student progress in the future, LA’s BEST could also institute the 
administration of a yearly self-efficacy survey to their participants as part of their enrollment 
procedures. This type of data collection would allow further study into the relationship between 
self-efficacy, self-regulation, and student outcomes from a longitudinal perspective. 

In summary and conclusion, it is encouraging to observe that LA’s BEST students are 
evaluating themselves in alignment with the outcome measures. According to contemporary 
literature, it is important to develop self-regulatory skills and become lifelong learners in the 21 st 
century. Thus, it is important that students learn self-evaluation strategies and self-monitoring 
skills, not only in oral communication and collaboration, but in all learning experiences. When 
students evaluate themselves, they are assessing what they know, what they do not know, and 
what they would like to know. Only in doing this can they begin to recognize their own strengths 
and weaknesses and keep on learning. 
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Appendix A: 

Propensity Weighting Process 
Propensity Scalar Formula 

Two logistic regression equations were used to estimate a propensity scalar of case. For 
cases from LA’s BEST Attendance Intensity Levels 1 and 2, the probability (PI) of being in 
Intensity Level 2 was used. To create a propensity scalar for cases from the third LA’s BEST 
attendance intensity level, the following formula was used: 

PS = ((1/1-P1)/P1) * P2 
Where: 

PS = Propensity scalar 

PI = The probability of being in Intensity Level 2 based on Logistic Regression 
Equation 1 (Intensity Levels 1 and 2). 

P2 = The probability of being in Intensity Level 2 based on Logistic Regression 
Equation 2 (Intensity Levels 3 and 2). 

The application of this formula had the effect of placing the estimates for the Level 3 
cases (obtained from Equation 2) on the same scale as the Level 1 and 2 estimates (obtained 
from Equation 1). The result was a unique propensity scalar that could balance background 
variables across all three intensity levels. 

Control variables with descriptive results before/after weighting 

As reported earlier, it was necessary in this study to control for existing differences in 
student background characteristics and other factors. Individual background variables used in 
the controlling process included parental education, gender, grade level, language 
classification, ethnicity, track code, and prior attendance in a non-LA’s BEST afterschool 
program. The date of student survey response was also used as a control due to an extended 
data collection period. We also controlled for site-level differences in the means of the total 
self-efficacy, oral communication, and collaboration composites and the total number of 4 th 
and 5 th grade students attending each program site. Means before and after weighting across 
the three LA’s BEST attendance levels are displayed for the continuous variables in 
Tables A1 and A2. As shown in Table Al, four of the five continuous variables had 
significant differences across the LA’s BEST attendance levels prior to the application of the 
weighting process. After the weighting was applied, no significant differences remained (see 
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Table A2). Percentages before and after weighting across the three LA’s BEST attendance 
levels are displayed for the categorical variables in Tables A3 and A4. As shown in 
Table A3, two of the five categorical variables had significant differences across the 
LA’s BEST attendance levels prior to the application of the weighting process. After the 
weighting was applied, no significant differences remained (see Table A4). 



Table At 

Continuous Background Variables: Means by LA’s BEST Attendance Level — Unweighted 







LA's BEST Attendance 
(2008-09) 




ANOVA 

results 


Un-weighted 

means 


1-99 days 
(n = 164) 


100-169 
days 
(n = 226) 


> 170 days 
(n = 307) 


F test 


Eta 


Sig. 


Survey date 


138.23 


111.08 


89.91 


10.884 


0.174 


0.000 


Site — Self-efficacy 


3.14 


3.16 


3.15 


0.569 


0.040 


0.566 


Site — Communication 


3.13 


3.16 


3.15 


4.356 


0.111 


0.013 


S ite — Coll aboration 


3.07 


3.09 


3.10 


4.086 


0.108 


0.017 


Site (4 th & 5 th ) — Attendance 


124.96 


105.61 


104.21 


9.335 


0.162 


0.000 


Table A2 














Continuous Background Variables: Means by LA's BEST Attendance Level- 


—After Weighting 








LA’s BEST Attendance 
(2008-09) 




ANOVA 

results 


Weighted 

means 


< 100 days 
(n = 164) 


100-169 

days 

(n = 226) 


> 170 days 
(n = 307) F test 


Eta 


Sig. 


Survey date 


128.18 


123.93 


132.78 


0.370 


0.033 


0.691 


Site — Self-efficacy 


3.15 


3.15 


3.15 


0.111 


0.018 


0.895 


Site — Communication 


3.15 


3.15 


3.15 


0.341 


0.031 


0.711 


S ite — C oil aboration 


3.08 


3.08 


3.09 


0.431 


0.035 


0.650 


Site (4 th & 5 th ) — Attendance 


116.28 


113.98 


111.07 


0.528 


0.039 


0.590 
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Table A3 



Categorical Background Variables: Percentages by LA's BEST Attendance Level — Unweighted 



Background variables 




LA’s BEST Attendance 
(2008-09) 




Chi-Square 

results 


<100 days 
(n = 164) 


100-169 

days 

(n = 226) 


> 170 days 
0 n = 307) 


Chi-SQ 

Value 


Cramers 

V 


Sig. 


Female 


60.4 


48.7 


59.6 


7.809 


0.106 


0.020 


Attendance in other ASP 


42.1 


34.1 


29.0 


8.206 


0.109 


0.017 


Ethnicity 








12.26 


0.094 


0.425 


Black 


9.1 


OO 

OO 


6.5 








Hispanic 


88.4 


85.0 


87.9 








Language classification 








9.028 


0.080 


0.172 


English only 


18.9 


31.0 


23.8 








IFEP 


16.5 


15.9 


16.9 








RFEP 


32.3 


23.5 


27.0 








LEP 


32.3 


29.6 


32.2 








Parent education 








8.443 


0.078 


0.207 


Declined/unknown 


33.5 


22.6 


29.3 








LT HS education 


32.9 


33.2 


33.6 








HS grad/no college 


19.5 


22.6 


20.8 








Had some college 


14.0 


21.7 


16.3 









Note. ASP = Afterschool program; HS = High school; IFEP = Initially fluent English proficient; LEP = Limited 
English proficiency; LT HS = Less than high school; RFEP = Redesignated fluent English proficient. 
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Table A4 



Categorical Background Variables: Percentages by LA's BEST attendance level — After weighting 







LA’s BEST Attendance 
(2008-09) 




Chi-Square 

results 


Background variables 


<100 days 
(n = 164) 


100-169 
days 
(n = 226) 


> 170 days 
(n = 307) 


Chi-SQ 

Value 


Cramers 

V 


Sig. 


Female 


53.7 


52.2 


57.0 


1.293 


0.043 


0.524 


Non-LA's BEST 
Attendance 


38.4 


37.6 


37.5 


0.044 


0.008 


0.978 


Ethnicity 








12.43 


0.094 


0.412 


Black 


9.8 


9.3 


8.1 








Hispanic 


87.1 


85.5 


83.4 








Language Classification 








0.596 


0.021 


0.996 


English Only 


23.2 


26.0 


24.4 








IFEP 


16.5 


15.9 


15.3 








RFEP 


29.3 


27.3 


28.3 








LEP 


31.1 


30.8 


31.9 








Parent Education 








3.976 


0.053 


0.680 


Declined/Unknown 


31.9 


24.8 


25.6 








LT HS education 


29.4 


35.0 


30.8 








HS grad/No college 


21.5 


21.2 


23.7 








Had some college 


17.2 


19.0 


19.8 









Note. HS = High school; IFEP = Initially fluent English proficient; LEP = Limited English proficiency; 
LT HS = Less than high school; RFEP = Redesignated fluent English proficient. 
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Appendix B: 
Survey Scales 



Self-efficacy 



Table B1 

Self-efficacy Multiple Group Model #1 



Factor 


Parcel 


Items 


Self-efficacy 


Academic_Positive (AcP) 


1,2,4 




Academic_Negative (AcN) 


3 




Self-Regulation_Positive (SRP) 


6, 7, 9, 10, 11, 13, 14 




Self-Regulation_Negative (SRN) 


5, 8, 12, 15 




Perseverance_Positive ( PeP) 


21,24, 25 




Perseverance_Negative (PeN) 


23,24 




Other_Positive (OP) 


16, 17, 19, 20 




Other_Negative (ON) 


18 



Table B2 

Self-efficacy Multiple Group Model #2 




Factor 


Parcel 


Items 


Self-efficacy 

Positive 


Academic_Positive (AcP) 
Self-Regulation_Positive (SRP) 
Perseverance_Positive (PeP) 
Other_Positive (OP) 


1,2,4 

6, 7, 9, 10, 11, 13, 14 
21,24, 25 
16, 17, 19, 20 


Self-efficacy 


Academic_Negative (AcN) 


3 


Negative 


Perseverance_Negative (PeN) 
Self-Regulation_Negative (SRN) 
Other_Negative (ON) 


23,24 
5, 8, 12, 15 
18 
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Table B3 
Self-efficacy 



Item # Item Subscale 

1 . I am good at learning math AcP 

2. I am good at learning science AcP 

3. I am not good at learning to read AcN 

4. I am good at learning to write AcP 

5. It is hard for me to finish my assignments on time SRN 

6. It is easy to concentrate when I am doing my schoolwork SRP 

7. It is easy for me to finish my homework on time SRP 

8. I am not good at taking notes during class SRN 

9. It is easy for me to use the library to get information for class assignments SRP 

10. I am good at using the computer to finish class assignments SRP 

11. I am good at organizing my schoolwork SRP 

12. I am not good at remembering information presented in class SRN 

13. I am good at remembering what I read in my schoolbooks SRP 

14. I am good at remembering what I read in my schoolbooks SRP 

15. I am not good at learning the skills needed for team sports SRN 

16. It is easy for me to live up to what my parents’ expect of me OP 

17. I am good at living up to what my teachers expect of me OP 

18. I am not good at living up to what I expect of myself ON 

19. It is easy for me to make friends OP 

20. I am good at keeping friends OP 

21. I like it when my schoolwork is challenging PeP 

22. It is hard for me to finish everything I start PeN 

23. Even when I try hard, I can't always finish my work PeN 

24. When something doesn't work the first time, I try again PeP 

25. It is easy to solve a problem if I try hard PeP 
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Oral Communication 



Table B4 

Oral Communication Model #1 

Factor Parcel Items 

Oral communication Interpersonal_Positive (IP) 2,7,15,21,22,23 

Interpersonal_Negative (IN) 4, 6, 8, 10, 14, 19, 20, 24 
Personal_Positive (PP) 5, 9, 11, 13, 16, 17 

Personal_Negative (PN) 1, 3, 12, 18 
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Table B5 

Oral Communication 



Item # 


Item 


Subscale 


1 . 


I don't talk right 


PN 


2. 


1 don't mind asking the teacher a question in class 


IP 


3. 


Sometimes I can’t figure out what to say 


PN 


4. 


It is harder for me to give a report in class than it is for most of the other kids 


IN 


5. 


I like the way I talk 


PP 


6. 


People sometimes finish my words for me 


IN 


7. 


I find it easy to talk to most everyone 


IP 


8. 


It is hard for me to talk to people 


IN 


9. 


1 don't worry about the way I talk 


PP 


10. 


1 don't find it easy to talk in front of other people 


IN 


11. 


It is easy for me to figure out what to say 


PP 


12. 


I’m afraid that kids will make fun of me when I talk 


PN 


13. 


Talking is easy for me 


PP 


14. 


Telling someone my name is hard for me 


IN 


15. 


I talk well with most everyone 


IP 


16. 


I would rather talk than write 


PP 


17. 


I like to talk 


PP 


18. 


I am not a good talker 


PN 


19. 


I wish I could talk like other children 


PN 


20. 


I let others talk for me 


IN 


21. 


Reading aloud in class is easy for me 


IP 


22. 


I am good at sharing my ideas during class 


IP 


23. 


I like to answer questions that people ask me 


IP 


24. 


I worry about asking questions during class 


IP 
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Collaboration 



Table B6 

Collaboration Multiple Group Model 

Factor Parcel Items 

Oral communication Peer Interaction (PI) 1, 6, 9, 10, 11, 14, 15, 16, 17, 18, 19, 20, 21, 22 

Process of Collaboration (Pr) 3, 4, 5, 8, 12 
Product of Collaboration (Pd) 2, 13 



Table B7 

Collaboration Items 



Item # 


Item 


Subscale 


1 . 


I listen to everyone in my group before I make a decision 


PI 


2. 


I enjoy my schoolwork more when I work with other students 


Pd 


3. 


The members of my group help explain things that I do not understand 


Pr 


4. 


The members of my group set goals for what we want to accomplish 


Pr 


5. 


I am more organized when I work in a group 


Pr 


6. 


I admit to the members of my group when I make a mistake 


PI 


7. 


I get upset when the kids in my group say bad things about my work 


PI 


8. 


I feel working in groups is a waste of time 


Pr 


9. 


I know that how I behave affects the other members of my group 


PI 


10. 


I listen to what everyone thinks before I make a decision 


PI 


11. 


I listen to what other people recommend and ask them questions 


PI 


12. 


When I work in a group, my work habits improve 


Pr 


13. 


I learn more information when I work with other students 


Pd 


14. 


I tell the other members of my group when 1 think they are doing a good job 


PI 


15. 


I try to give helpful suggestions to the members of my group 


PI 


16. 


I treat the other members of my group with respect 


PI 


17. 


I am a great listener 


PI 


18. 


I try to understand what the other members of my group are saying 


PI 


19. 


1 do not stay angry or upset for very long 


PI 


20. 


I would rather say what I think then listen to my classmates 


PI 


21. 


I try to focus on solving problems rather than who is to blame 


PI 


22. 


I look for answers that meet everyone’s needs 


PI 
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